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ABSTRACT 



A research project evaluated the effectiveness of an 
Electronic Performance Support System (EPSS) . The population consisted of 100 
telephone operators in the sales and service department of a Dutch bank. The 
research ascertained that the EPSS was the support available to operators 
through a large screen monitor at his or her disposal: a tool environment (to 
process the data) and an information environment (quick access to up-to-date 
information) . A pretest was preceded by eight semistructured interviews from 
which researchers learned more about the work conditions in the section, 
working with the tool, and use of the information part in the support system. 
During observations, observers sat next to working telephone operators. The 
variables used in this measurement were conversation time and information 
sources used. A questionnaire asked 24 operators about their motivation, 
work, and tool and information use. Pretest results suggested that 
experienced operators used the information component more frequently than 
novices and that operators were satisfied with the tool, although 
administrative improvements were suggested. The real improvement would be 
made in the information component. Posttest results indicated that operators 
felt a deficiency of standardization in the work. They were very satisfied 
with the new information component, which was used more frequently than the 
old. It did not reduce the search for and use of other information sources. 
(Contains 10 references.) (YLB) 
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The early eighties witnessed the large-scale introduction of computers to most 
areas of commerce, industry and education. Inevitably this created considerable 
change in work procedures and attitudes. In the field of performance technology 
several electronic support systems have been introduced. Nowadays keywords 
are "just in time training" and "learning by doing". Computer support and 
electronic job-aids are approximating the master in the master and journeyman 
relationship which existed in former times. This chapter is a report of a project in 
the field of Electronic Performance Support Systems (EPSS). Firstly, the focus 
is on a definition of a composition of an EPSS. Secondly, the theoretical 
advantages of EPSS use are stated. Finally the findings of a research project are 
discussed. The project evaluates and improves an existing EPSS for telephone 
operators. 

An EPSS is an integrated computerised environment that supports and 
occasionally monitors employees while they perform their jobs. In general an 
EPSS is made up of the following four components (see figure 1): tools (to 
perform the job), information (needed to do the job correctly), advice (for the 
difficult parts of the job) and training (to extend the employees' knowledge and 
skills). The system is a substitute for, or enhances the support of, a master. 
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Figure 1: 




In the literature several possible advantages of the use of an EPSS are reported. 
These are assumptions of the possible advantages of EPSS use in practice. The 
first advantage relates to on-the-job training which leads to high transfer, no 
need to leave the workplace, and more-active learning processes (Bastiaens, 
1995). Probably the most important advantage is the immediate access to 
information, training and advice (Gery 1989, 1991). The just-in-time access to 
information leads to an extension of the employees' long-term memory and a 
reduction of the working load memory (Law, 1994). Having continuous access 
to training means a reduction of formal training in advance of actual task 
performance. Because employees can constantly consult the advice part of the 
EPSS it is expected that the need for a supervisor's guidance will be less. This 
has the additional advantage that the responsibility focus shifts from trainer and 
training program to the individual’s learning needs (Gery, 1991). Moreover, 
performance support can be important for the employees’ self-management, or 
for the guidance of self-directed teams (Bramer & Senbatta, 1 993) and as such 
has the potential to improve the worker's productivity (Raybould, 1990; 1991). 
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Problem Statement 

In the previous section assumptions on the advantage of EPSS have been 
mentioned but little has been empirically proven. This research project attempts 
to evaluate the effectiveness of an EPSS. A few disadvantages were found in the 
literature that could affect the effectiveness of an EPSS. These disadvantages fall 
into three categories. 

The first category is related to the learning process. "Just-in time" 
training at the workplace, providing employees with small task-oriented training 
granules and employees taking control of their own learning process can create 
problems. Clark (1992) argues that employees may fail to build a unified picture 
of their job when they have to extract information from an EPSS. Several small 
information parts will create a fragmented knowledge base. Novices especially 
need a high level overview of the content to relate details of training. Clark has 
doubts about the learner control in an EPSS and she illustrates it with the 
research of Milheim and Martin (1991) which indeed proves that learner control 
is not as effective as instructional control. This means leaving learners to their 
own devices and let them make their own decissions is not as effective as let the 
computer environment taking over control and provide a predefined and 
structured route. 

The second category involves problems related to innovation. It is 
expected that the introduction and implementation of EPSS will create 
resistance. Employees are not likely to give up working in “the old way”. Even 
if they are willing to try a new method there is the problem of work pressure. 
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Employees may believe they do not have the time to use the training support 
system. 

The third category is related to the performance support and the work 
that has to be done. Is the support adequate? Will "just-in time" support “de- 
skill” workers? Will it 'demotivate' workers? Or will it automatize the low level 
tasks and as a result give workers the opportunity to concentrate on high level 
tasks (Carr, 1992)? 

While these issues provide many questions, the research reported in this chapter 
focuses on the questions stated in the section “research questions”. 

The Setting 

The research project was executed in cooperation with a large Dutch banking 
organisation which provides banking services and advice by telephone. Their 
telephone operators are extensively trained for this task. As far as the training 
and experience are concerned, two types of telephone operators are 
distinguished. Firstly, there are the experienced operators who have the skills to 
give advice for all the products the bank has to offer to their clients. Secondly, 
there are novice operators who start with just a few products. Novice operators 
combine training and working and extend their knowledge and skills during a 
period of six months. After six months training the novice workers are 
considered sufficiently trained to give advice on all the products. 

To give advice and information to clients and also to process their 
clients’ data, the telephone operators are supported by a personal computer that 
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is attached to a mainframe. Each operator has a large screen monitor at his or her 
disposal. On this monitor it is possible to use a tool environment (to process the 
data) and an information environment (quick access to up-to-date information) 
simultaneously. Although a computerbased training part was not included we 
ascertained these environments together to be an EPSS. 

Research questions 

The main purpose of the research study is to evaluate the existing 
support environment and if possible improve it. The research questions are as 
follows: 1) What is the effect of the existing support environment on the 

performance and learning of the telephone operators? 

2) How can an the existing support environment be improved and 
raise the effectiveness of the support environment? 

3) "What is the effect of the optimisation on the performance of 
the telephone operators?" 

The following hypotheses were developed: 

Hypothesis 1 relates to differences between novice and experienced 
operators: There is a difference between novice and experienced operators. 
Novice operators need more information and different information than the more 
experienced operators. 

Hypothesis 2 relates to the support of the information component: It is 
expected that novice operators will appreciate the information component more. 
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because of a more serious need for information. Lack of information will affect 
their performance. 

Hypothesis 3 relates to the motivation of the operators: Novice operators 
have a higher motivation but are more insecure about their own performance 
than experienced operators. For this reason it is expected that novices will have a 
higher score on the willingness to innovate. 

Hypothesis 4 relates to the treatment: It is expected that following the 
optimisation the new information component will be used more often. It is also 
expected that a new information component will reduce the use of other 
information sources. Finally, the assumptions and disadvantages stated in the 
literature are compared with the effects related to learning, innovation, support 
and work in this research. 

Methodology 

To evaluate the support environment the methodology of the one group pre-test 
post-test design was used (figure 2). This design exists of a pre-test, a treatment 
and a post-test. For practical reasons it was not possible to use a control group. 

The pre-test provides an insight into the information use of both 
experienced and novice operators. To establish whether there is an improvement 
after the treatment, a comparison of the results of the pre-test with the results of 
the post-test are made. For this purpose the post-test was a replica of the pre-test. 




To improve the reliability of the findings the method of triangulation is used. 
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This means that the data is collected by various instruments for datacollection 
and that the results are based on more then one instrument. 

The population consisted of 100 employees of the sales and service 
department savings-accounts. The difference in working experience, age and 
educational background was measured and the test groups were measured for 
homogeneity. The outcomes showed no significant difference between the two 
groups. We concluded there was no difference between the groups. 



01 02 03 X 04 05 06 



01 Interviews 04 Interviews X Treatment 

02 Observations 05 Observations 

03 Questionnaire 06 Questionnaire 



Figure 2 \_ Overview of the methodology. 

The theoretical construct 

The conceptual framework included constructs used in an earlier research project 
(Bastiaens, Nijhof, Streumer & Abma, 1997). The constructs describe the 
variables that exert an influence on the EPSS (table 1). Working, treatment and 
background are general variables. The constructs' tool and information together 
are the support environment. The tool environment is used to process the data 
and the information environment is used to quickly access information which is 
needed to perform. Both tools support the operators while they are doing their 




job. 



9 

Next to this, attitude towards work and performance are important in measuring 
the work context. To understand the knowledge, skills and attitudes learned in 
the introductory course a number of variables were identified and taken into 
consideration. The course may influence the use of the tool and the information 
environment. 

Variables were derived from the constructs from which items were 
developed for the questionnaires given to the operators. 



Constructs 


Variables 


Work 

attitude towards work 
performance 


motivation, self-confidence 
independence 


Tool 

communication 

help 

menu 


interface, technical construct, technical realization 
content, communication 
service and advice 


Information 

communication 

information 

information (other origin) 


interface, technical construct, technical realization 
service and advice, usefulness, structure 
information 


Course 

preparation on performance 


Knowledge, skills, attitude 


Background 

Personal 

Experience 

attitude towards innovation 


age, gender 

educational background, working and computer 
working with new technology 



Table 1 : A list of constructs 



Results of the pre-test 

The pre-test was preceded by eight semi-structured interviews. From the 
interviews the researchers wanted to learn more about the work conditions in the 
section, working with the tool and the use of the information part in the support 
system. While we cannot provide all of the results in this chapter the interviews 
showed that the operators in general were very satisfied with their job. They 
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liked both their jobs and the tool. However, they were not as satisfied with the 
information part in the support system. This resulted in a low average use of the 
information component which was described by obsolete information and a dull 
interface with too many levels in a hierarchical structure. It was easier to consult 
a colleague or to use their own notes (which were used very often as an aid to 
quick reference). 

The second instrument used in the data collection were the observations. 
To collect these data an observer sat next to the telephone operator when he/she 
was working. The variables used in this measurement were conversation time 
and the information sources used. The independent variables were gender and 
experience. Exactly 150 telephone calls were observed. In 37 conversations the 
ten operators used an information source. Table 2 shows the categories of 
employees and the information sources used. 



Employee 


calls 


use information 
component 


use other information sources 


total 


paper verba! . 


experienced 


60 (100%) 


3 (5.0 %) 


8 (13.4%) 


1(1.1%) 


12 (20%) 


novice 


90(100%) 


0 (0.0 %) 


19 (2-1.1%) 


6 (6.7 %) 


25 (27.8%) 


total 


150 


3 


27 


7 


37 



Table 2: Observed use of information sources in the pre-test. 

For information use, experienced employees consulted information sources in 
20% of the telephone calls. For novices it was 27.8 % of all calls. Interestingly, 
novices do not use the information component at all. The general conclusion 
derived from table 2 is that the information component in the support system 
was not used very often. The operators more often used another source. A closer 
look at those sources showed that they asked their colleagues for information 
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seven times, used written sources such as folders, brochures and handbooks for 
27 times. 3 

Regarding the first hypothesis which is related to differences between 
novices and experienced operators, the observations showed a difference 
between novice operators and experienced operators. Although the two groups 
used other information sources, the difference in conversation time was not 
significant according to the results of a t-test. The average time for experienced 
operators was 2.28 minutes (s.d. 1.81), for novices it was 2.52 minutes (s.d. 
2.31). 

The third instrument to collect the data was the questionnaire. In the 
questionnaire 24 operators were asked about their motivation, work, tool and 
information use. First their motivation was looked at. 

The hypothesis that there is a difference in motivation between novice 
(nl=8) and experienced operators (n2=15) was not supported. Although we 
expected novices to have a higher motivation level related to their work, this was 

not confirmed (Mann- Whitney test, U=55.0, 7? 1= 1 1 .38,^2= 12.33). 

The hypothesis that novices are more insecure and that experienced 
operators are more satisfied about their performance was also not significant 

(Mann- Whitney test, U= 53.0, 7? ,=11.13, ,R 2 = 12.47). Both types of operators 
perceive their performance to be high. 

The hypothesis that novices score higher on innovation willingness is 
also not significant. Both groups scored the same on innovation willingness 
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(Mann- Whitney test, U= 63.5, /?,= 12.56, ,R 2 = 12.47). There is no statistical 
difference in how they experience new technology. 

Important for the further improvement of the performance support 
system they were asked about the technical realisation (errors in software, 
waiting time, etc) of the tool. When the telephone operators were asked their 
opinion about the tool it appeared that that the two groups had the same opinion 
about the technical realisation, the help content (procedural information) and the 
communication with the help content. Table 3 shows the results. 



Pre-test 


Post-test 








U 


p* 




*2 


U 


p* 


technical realisation 


15.19 


11.16 


42.5 


p>.05 


13.12 


14.82 


79.5 


p>.05 


help content 


8.57 


12.21 


32.0 


p>.05 


11.12 


16.68 


53.5 


p>.05 


communication 


10.64 


11.18 


46.5 


p>.05 


12.54 


15.36 


72.0 


p>.05 



* double tailed 



Table 3: Opinion about the tool component. /?,=meanrank novice R mean rank 
experienced. 



The following results are related to the information component. The operators 
were asked about the technical realisation, accuracy of the information and the 
use of other information sources. Table 4 shows no significant differences 
between the two groups. 

When we looked at the use of the information component we saw that 
experienced operators use the information component significantly more often 
(Mann- Whitney test, n,= 8, n 2 = 16, U= 26.5, p<.005). 
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Pre-test 








Post-test 












*2 


U 


p* 




*2 


U 


P* 


technical realisation 


14.13 


11.69 


51.0 


p>.05 


12.23 


14.77 


68.0 


p>.05 


accuracy of information 


10.06 


13.72 


44.5 


p>.05 


11.31 


15.69 


56.0 


p>.05 


use of other information 
sources 


10.13 


13.69 


45.0 


p>.05 


11.31 


15.69 


56.0 


p>.05 



* double tailed 



Table 4: Opinion about the information component in the pre-test and post-test. R ,= mean rank 
novices R 2 ~ mean rank experienced. 



So the hypothesis that novices use the information component more often than 
experienced operators was rejected. 

In general the following can be concluded. For most hypotheses the pre- 
test shows no difference between experienced and novice users. It is surprising 
that experienced operators use the information component more frequently than 
novices. All the employees provided interesting suggestions about how to 
improve the system. These were taken into consideration when a new support 
system was constructed. 

Recommendations for the construction of a new support system 

In general the findings show that the operators are satisfied with the tool, 
although administrative improvements were suggested. The real improvement 
has to be made in the information component. The data show that experienced 
users actually do use the information component more often than novices. 

From the interviews we learned that they use it as a reference book simply 
because not every detail of a product can be remembered. On the other hand 
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novices use other information sources more often, even for factual knowledge 
(where experienced users use the system). The novices told us that they do not 
know how to use the system because they are confused by its hierarchical 
design. They find it hard enough to advise a client and operate the telephone 
system without using an information system that is not user friendly. 

It is also interesting that they need another kind of information. The 
information they require is not solely knowledge of the facts but also 
instructions about how to do the job. This sort of procedural information was not 
yet available in the system. The evaluation showed that it is important that users 
can trust the information and that therefore information has to be relevant, 
accurate and up-to-date. The final important recommendation is that the search 
and use of the information component has to be an integral part of the overall 
performance. That means the operators have to integrate the information search 
as a skill in their performance. 

The recommendations require a context and a task analysis before a new 
information component can be constructed. Also some technical conditions have 
to be taken into account. 

In figure 3 the schematic design of the traditional information component and 
the constructed new information system with a maximum of three layers and 



help questions (Bastiaens, 1994) are shown. 



Before 



After 



Products 

* giro savings 

* youth savings 
Interest 

* overview 

* calculation of interest 



Savings (general) 






* automatically 
Products 


What? 


How? 


* giro savings 


What? 


How? 


* youth savings 
Data 


What? 


How? 



Various 



* address and phone 
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Figure 3: Outline of the system before and after its redesign. 



Results on the post-test 

The post-test consisted of the same instruments and identical variables as used 
for the pre-test, so semi-structured interviews (n=8) were used first. As in the 
pre-test the operators in general are very satisfied with their work. One important 
item they told the researchers the second time was that there was a deficiency of 
standardisation in the work. Although the EPSS provides some standards to 
process the data, operators want more uniform procedures. The operators are 
also less positive about the communication process in the division. New rules, 
products and procedures are not communicated as quickly as they should be. 
They are very satisfied with the new information component. They use the 
information component because of the new user-friendly structure and the up-to- 
date information. On the other hand, because of a lack of time at the workplace 
they would like to explore the component off the job. They miss a personal 
scratch-pad in the system and the possibility to structure the interface 
themselves. 

The next part of the post-test were the observations. Again 150 calls 
were observed. In table 5 the results are shown. 



Employee 


calls 


use information 
component 


use other information sources 


total 


paper verbal 


experienced 


60(100%) 


4 (6.7 %) 


2 ( 3.3 %) 


3(5.0%) 


9 ( 15.0%) 


novice 


90(100%) 


7 (7.8 %) 


14(15.6%) 


9(10.0%) 


30(33.3%) 


total 


150 


11 


16 


12 


39 
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Table 5: Observed use of information sources in the post-test. 

The table shows that the information system is used 1 1 times, other information 
sources 28 times. The total use of information is 39 times. A closer look at the 
data indicates that the operators consult colleagues 12 times and use brochures 
and handbooks 16 times. The observations in the post-test again showed a non- 
significant difference (t-test) in the average conversation time between novice 
operators (M, =2. 25, s.d.=2.19) and experienced operators.(M 2 = 2.19, s.d.=1.48). 

The last part of the post-test was the questionnaire. The questionnaire 
was completed by 27 operators. From table 3 it appears that no significant 
differences were found when the operators were again asked their opinion about 
the tool (technical realisation, the content of the help and the communication 
with the help component). 

The results related to the information component (table 4) show that the 
operators’ opinion about the technical realisation was significantly different 
from their opinion in the pre-test. The accuracy of the information in the 
information component is appreciated more by experienced operators. The 
hypothesis related to the support of the information component is not supported 
by the data. It was expected that novice operators would appreciate the 
information component more, because of a more serious need for information. 

In the use of other information sources no significant differences were found 



between the two groups. 
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Comparison between pre-test and post-test 

After the development of a new information component the hypothesis is that 
the new information component is used more frequently than the old one. This 
happens to be true ( % 2 , df= 1, p< .05). Novices especially, use the information 
component more often. Their score on the pre-test was 0 and after the treatment 
on the post-test their score was 7. This is significantly different (% 2 , df= 1, p< 
.05). 

It was also expected that the new information component would reduce 
the search for, and use of, other information sources. This turned out to be untrue 
and, in fact, the consultation between colleagues has grown. This is not a 
significant growth (% 2 , df= 1, p>.05). On the other hand the use of written 
information sources did drop significantly (y 2 , df= 1, p<.05). 

The hypothesis that the new information component supports the 
operators better than the old component is not affirmed. There is a difference in 
the mean scores in the advantage of the new component but the difference is not 

significant (Mann- Whitney test, /?,= 22.94, R 2 = 21.85, U= 250.5, p>.05). 
Although the users think that the new support is an improvement, this is not 
reflected in the comparison between the pre-test and post-test scores. 

When an evaluation is made on the theoretical assumptions, the projects 
show that the support environment leads to training on the job. The reduction of 
formal learning is high. In this organisation formal classroom training is reduced 
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from one month to six days. The support environment also leads to a reduction 
of the working load memory. The assumption of resistance was not observed but 
it was shown that this is hard for the operators to give up old ways of doing their 
job. The automation of tasks and the easy consultation of the support 
environment led to an extension of tasks. The results showed that before it was 
‘demotivating’ when operators had to wait too long before they could broaden 
their knowledge about other products. 

Conclusions 

The results show that the use of the information component was initially very 
low. From that a lesson can be learned: an information component has to be up- 
to-date, complete and its use has to be an integral part of the performance. Our 
advice was to hire one employee to update the information on a daily basis. This 
person should not only put in official information but also collect notes and 
small job aids. Another suggestion is the integration of the use of the support 
system in the formal learning process. New employees should be taught to work 
with the support environment. They should be given the time and opportunity to 
explore the support environment before setting them to work. 

Before designing a support system developers have to keep in mind that 
the people who use the system are very diverse. Potential users have their own 
needs. They should be given the opportunity to change the interface in their own 
way, integrate a notepad and provide different ways to search for information. 

Although our example is not a complete EPSS as is stated in the first 
section, it can be concluded that the developed support system in this situation 
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meets the needs of the company. Developing a complete EPSS in every situation 
is not necessary. In this project some advantages of electronic support have been 
supported by data but serious disadvantages have also been identified. Further 
research in this field is required to understand not only the design of effective 
EPSS but more importantly in to get an insight in the analysis of the 
organisation, the performance and the workers. However performance support 
has a brilliant future, it will inform and train employees and help them to 
perform better in a shorter time. 
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